Shader editor and compiler

ABSTRACT

A software application that allows users to create shader definitions using an intuitive graphical user interface. The invention is characterized by an intuitive user interface with a streamlined workflow that non-technical users (artists) are able to understand and utilize without the need to become technically proficient in computer software expression. This interface allows the user to create a shader graphically without writing code, by wiring predefined filter, assembly and property primitives together into a dynamic, directed wire graph. A high level code definer writes an XML (for example) script for the defined wire graph. The output produced by the system is translated into a shader program implemented in a format that is recognized by some other rendering system. Typically this means a shader program represented as a text file containing code in some specific shader language. The system is able to generate output targeted for specific platforms based on a single graphical representation of the shader program created by the user.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is related to and takes priority from U.S.Provisional Patent Application Ser. No. 60/448,316, filed Feb. 18, 2003,entitled “Shader Editor and Compiler, commonly owned with the assigneeof the present invention, the entire contents of which are expresslyincorporated herein by reference.

FIELD OF THE INVENTION

The present invention is directed to systems and methods for graphicalimage rendering and compositing and, more particularly, to systems andmethods for developing programmable shaders that may be implemented intoreal-time rendering systems without regard to the rendering system'sprogramming language.

BACKGROUND OF THE INVENTION

Historically, graphics rendering systems have typically employed theconcept of a shader or material to describe color, and variousattributes of a color, of a region on the surface of an object. Shadersare commonly implemented as procedural plug-ins which allow differentmethods of computing a surface color to be substituted, into aprocedure, based on the desired affect. A shader plug-in is typicallyimplemented as a program written in a specific shader language,particular to the rendering system that will utilize it.

In the past, only off-line rendering systems allowed for programmableshader descriptions, while real-time rendering systems used a fixedlighting algorithm. Recent advances in graphics hardware have made itpossible to incorporate programmable shaders into real-time renderingsystem environments and have lead to the emergence of a number ofdifferent programming languages with which to write these shaders.

Contemporary software applications are, by and large, developed withhigh level programming languages, such as “C”, and the like. However, inorder to create complex visual effects, artists have had to rely onhighly restrictive assembly language programs that are written directlyto graphics hardware. Accordingly, an artist is required to spend aconsiderable amount of time and effort in programming his or hercomputer; time and effort that could be better spent in creatingvisually exciting graphics images.

Although we live in a world in which professional, high-end graphics andvisual communication are common place (both on television and in films)it is often quite difficult for an artist, lacking significantprogramming skills and training, to create such graphics simply andeasily. In particular, there are no tools currently available that allowan ordinary user to create various shader definitions utilizing asimple, intuitive graphical user interface in order to create polished,high-impact visual media in a timely or cost-effective manner. Moreover,despite the broad appeal and large in information band width inherent inwell-produced graphical presentations, there are no systems or methodsin the current marketplace, even for professional graphic artists, thatefficiently convert various programmable shader descriptions into shaderprograms for real-time hardware.

SUMMARY OF THE INVENTION

In particular, the invention is directed to a software application(termed Baku herein for purposes of convenience) that allows users tocreate shader definitions using an intuitive graphical user interface.The invention is characterized by certain particular features; anintuitive user interface with a streamlined workflow that non-technicalusers (artists) are able to understand and utilize without the need tobecome technically proficient in computer software expression. Thisinterface allows the user to create a shader graphically without writingcode.

A user is able to create an infinite number of unique shaders. This isdifferent from simply changing the parameters to a single shader. Theuser will be able to actually define new shading algorithms utilizingBaku's graphical representation. The output produced by Baku ischaracterized as a shader program implemented in a format that isrecognized by some other rendering system. Typically this means a shaderprogram represented as a text file containing code in some specificshader language. Baku is able to generate output targeted for specificplatforms based on a single graphical representation of the shaderprogram created by the user. Target output formats can be added asneeded, however for ease of explanation, the invention specificationfocus will be placed on shader programs for real-time hardware such asprograms in the HLSL or Cg language format.

DESCRIPTION OF THE DRAWINGS

These and other features, aspects and advantages of the presentinvention will be more fully understood when considered with respect tothe following specification, appended claims and accompanying drawings,wherein:

FIG. 1 is a simplified, semi-schematic representation of examples offilters, properties and assemblies in accord with the present invention;

FIG. 2 is a simplified, semi-schematic representation of a materialassembly structure;

FIG. 3 is a simplified, semi-schematic representation of vertex andpixel shader wire graphs;

FIG. 4 is . . .

DESCRIPTION OF THE INVENTION

Specifically, the invention is directed to a software application thatallows graphics artists to create shader definitions using a simple,intuitive graphical user interface. The invention is characterized bycertain particular features; and intuitive user interface with astreamlined workflow that non-technical users, such as graphics artists,are able to understand and utilize without the need to becometechnically proficient in computer software programming. This interfaceallows the user to create a shader graphically, modify all of itsparametric definitions, and view the resulting object, all withoutwriting code.

A user is able to create an infinite number of unique shaders. It shouldbe understood that this is different from simply changing one or more ofmultiple parameters to a single shader. In accordance with theinvention, a user is able to actually define new shading algorithmsutilizing the invention's graphical representation. The output of thesystem is characterized as a shader program implemented in a programthat is recognized by some other rendering system. Typically, this mightbe characterized as a shader program, represented as a text file, whichcontains code in some specific shader language, with the generatedoutput targeted for specific platforms based on a single graphicalrepresentation of the shader program created by the user.

In particular, a shader, editor and compiler, in accord with the presentinvention, implements the paradigm of a wire graph. In terms of theinvention, the wire graph is a directed acyclic graph (DAG) of nodesthat are connected by what are commonly termed wires. Wires connect tonodes and have a direction that represents the direction of the flow ofdata. When discussing two nodes, connected by a wire, one refers to theupstream node as the source and the downstream node as the sync. Theuser creates nodes and uses wires to establish connections betweennodes, the end result being a graphical representation of a model thatdescribes functional elements and how they are evaluated. In otherwords, the wire graph forms a model of a shader editor program in amanner similar to a computer program data model and flow diagram.

As will be understood by those having skill in the art, directed acyclicgraphs are directed graphs with no cycles. Directed acyclic graphs arepart tree, part graph and have many applications, such as thoseinvolving precedence among “events”. Many problems in graphicaldescription become relatively simpler, when utilizing directed acyclicgraphs, particularly expression tree evaluation because of DAG's abilityto be topologically sorted using depth-first search.

By way of background, and with reference to the exemplary embodiment ofFIG. 1, there are, typically, three types of graph nodes; assemblynodes, filter nodes and property nodes. An assembly node 10 might beappropriately described as a grouping of graph elements into a singleunit for organizational purposes. A particular assembly node might haveany number of inputs and/or outputs that are “visible” to theenvironment outside of the assembly node. Additionally, an assembly 10might also contain other types of nodes as internal elements which maynot necessarily be visible to the environment outside the assembly.Characteristically, assemblies may contain filters, properties, and evenother, nested assemblies which might themselves incorporate variousother graph nodes.

The elements of an assembly form what is termed a subgraph (which itselfmay or may not have its various inputs and outputs completely connectedto one another). Inputs and outputs of the elements in this containedsubgraph can be connected to the inputs/outputs of the assembly, as wellas to the inputs/outputs of contained nodes. As will be described ingreater detail below, this property allows a subgraph to be collapsedinto a single representational unit, with selected components exposed tothe external environment as assembly inputs and outputs. As mentionedabove, assemblies may be nested, such that one assembly is able tocontain an other assembly as a complete internal element.

A filter node (or “filter”) 12 includes a collection of inputs andoutputs (typically a single output) and represents a function thatcomputes an output value based on input values or properties. It shouldbe further understood that a filter input is able to be wired tofunction as either a source or a sync. When wired as a sync, a wireconnection provides a value or parameter from some other node to thefilter input, where the value or parameter is used by the filter'scomputation function. Conversely, when a filter's input is wired as asource, the filter does not perform its computational function on theinput value, but rather acts as a simple pass-through, thereby providingthe value of an input to a downstream element as though it were anadditional output. Necessarily, however, the primary outfit of a filtermay only be wired as a source.

A property node, indicated generally at 14, typically represents asingle data value, or parameter, such as height, a radius, an angle, ageometry definition, and the like. A property node may be used as aninput to a filter, or it may stand alone as an element in asub-assembly, assembly, or the graph. A property node may be wired aseither a source or a sync, but, irrespective of whether it is wired as async, it always contains a value that it can provide to any clientelement to which it is wired as a source. This particular containedvalue is commonly referred to as a “seed value”.

In the context of the exemplary embodiment of FIG. 1, a bent cylinderassembly 10 can be seen as comprising a number of graph element nodeswhich are coupled together in order to define a bent cylinder. The bentcylinder assembly 10 outputs a geometry 15 based upon input propertydefinitions of angle 16, radius 17, and height 18. Although angle,radius and height are properties defining the inputs of the bendcylinder assembly 10, it will be seen that the radius 17 and height 18properties are wired as sources to the cylinder filter 12, whichmathematically computes a cylinder geometry based upon these inputs.

The angle property 16 is wired as a source to a bend filter 13 whichalso receives the geometry property from the cylinder filter 12. Thebend filter 13 applies the angle property to the cylinder geometry andoutputs a bent cylinder 15. The “bent cylinder assembly” is thus thecombination of the various filters and properties that make up a “bentcylinder” object.

In accordance with the invention, a wire graph, or wire graph portion,can be implemented as an active functional element or a passive datastructure. Within an active functional wire graph, a filter (orassembly) is able to query its input properties in order to retrievetheir values. In turn, this causes property elements to evaluate theirown wires (if indeed they are wired) which may, in turn, triggerevaluation of other filters, in accordance with the tree-like structureof the directed acyclic graph nature of the system. Once inputproperties are queried and retrieved, the filter combines the value ofits input properties in order to produce an output that is madeavailable to client elements which are, in turn, wired downstream fromthe filter.

Conversely, a passive wire graph is simply and purely a representationaldata structure that depicts a functional relationship between variouselements but is not active in the computation of those functions. Forexample, a shader graph in the context of the invention is a portion ofthe wire graph that is passive and purely representational. The shadergraph represents a shader program by depicting how inputs are combinedmathematically in order to produce an output surface color, therebycreating an abstract model of a shader program. The shader graph is notdirectly evaluated by the invention but rather interpreted and convertedinto a shader program suitable for a particular output target platform.The output shader program is then able to run on its own on the hosttarget platform.

The system also allows creation of a seen wire graph in order to definean environment in which shaders (or shader results) are previewed. Aseen wire graph is an active, functional wire graph that is evaluated insoftware as opposed to being interpreted into a shader program in sometarget language. Seen elements, such as geometry, lights, cameras, andthe like, are implemented by filters that combine values from theirinputs to produce an output, as described above. Utilizing the exampleof FIG. 1, a geometry-producing filter such as a cylinder may have aradius and height input that the filter uses to generate cylindergeometry of the appropriate size. A seen is rendered by evaluating thisand other filters in order to obtain the geometry, lights, cameraangles, and materials suitable for the rendered view.

Turning now to FIG. 2, a “material” in The system is structured toinclude the purely representational shader wire graph in addition tocomponents in the scene wire graph. The shader wire graph isencapsulated in an assembly called the shader assembly. This shaderassembly is placed into another assembly that represents the material(the material assembly) along with a material filter. Each materialcontains properties that specify values corresponding to variables inthe shader program. Several materials may refer to the same shaderassembly, each with its own set of properties that correspond tovariables in the shader. Thus, a shader assembly essentially defines aprogram while the material assembly provides the data (values forparameters) for that program. Just as a program can be run withdifferent sets of data, a shader assembly can be used with differentmaterials.

The properties of the material assembly are available to be wired toother parts of the scene wire graph. This portion of the wire graph isnot interpreted into a shader program but is instead run in softwaredirectly. For example, a subgraph of filters could be wired together toproduce a color used as a value for a variable provided to the shaderprogram. However, from the point of view of the shader program, thisvalue is static because it is not computed as part of the computation ofthe shader program.

As can be understood from FIG. 3, the shader wire graph uses the sameelements as a scene wire graph (filters, properties, and assemblies) tobuild a graphical model of a shader program. This model is focused onthe current needs of real-time hardware rendering but is also applicableto software-based offline rendering. The shader graph is made up of twosubgraphs. One represents the per-vertex portion of the program and onerepresents the per-pixel part of the program. The per-vertex programgets vertex data as input and produces a collection of output values.The per-pixel program gets the output of the vertex program as input andproduces an output surface color. Both offline software rendering andreal-time hardware rendering have these two stages as part of theirprocess of rendering. First vertices are processed (the vertex program)and then the result is used to rasterize surface elements (oftentriangles) to produce surface fragments to be shaded by the pixelprogram.

Both the pixel and vertex program graphs have some common elements.Properties can be created to represent input variables to the shaderprogram. There are two types of variables: varying inputs, such as“Vertex Position” in FIG. 3, and uniform inputs, such as “ProjectionTM”. Varying inputs change as the shader program process each elementwhile uniform inputs are constant as an object is rendered. For example,the vertex data (varying input) passed to a vertex program varies aseach vertex is processed while the world transformation matrix (uniforminput) is constant for each vertex processed. The varying inputs to apixel program are the outputs of the vertex program.

A shader output filter, such as the “Pixel Shader Output” of FIG. 3,represents the collection of data that is output from a shader program.Each input property of this filter represents a component that is outputfrom the shader program. A vertex program must at least output aposition, but can also output other elements in addition. A pixelprogram always just outputs a color that represents the color of thesurface being shaded.

The input variable properties are wired to filters (“Transform Point” ofFIG. 3) to form a graph that is ultimately connected to the shader'soutput filter. Each filter represents a function or operation. Thiscould be a mathematical operation like multiplication, addition, or adot product. Or it could represent other types of operations such as afunction that looks up and retrieves a pixel from a texture map. Thereare also filters to convert data types or extract a component of a datatype.

The user creates these elements and wires them together to form a shadergraph for the pixel and vertex shader programs. The vertex and pixelshader graphs appear as unconnected graphs, although the varying inputproperties of the pixel shader refer to the same data elements as theoutputs of the vertex shader. Portions of these shader graphs can bebundled up into an assembly and collapsed to appear as a single element.This has two purposes: to reduce visual complexity of the graph so thegraph can be better visualized and to create a computational unit thatcan be reused in other shaders through a process called templatization.

A portion of a wire graph encapsulated in an assembly can be templatizedto allow it to be reused. This process records the structure of thecontents of the assembly so it can be used as a template to createanother instance of that assembly. Entire shader graphs can betemplatized or just subgraphs.

Using the process of collapsing subgraphs into single units (assemblies)and templatizing these units for reuse, effectively creates differentlevels of complexity at which a user can choose to work because a usercan use these pre-made units as building blocks for their own shadergraphs, as shown in the exemplary embodiment of FIG. 4. This allows anartist with little technical knowledge to work at a high level usingpre-made assemblies that encapsulate the details (and hide them from theuser) or a more technical user to work with low level filters forcomplete control over the shader program. Also, an artist with a smallamount of technical knowledge can work with medium level assemblies thatstrike a compromise between hiding complexity and providing more controlso these users can create a wide variety of shaders without programming.

Filters represent an operation or mathematical function of some sort.They store this representation in a code fragment written in anintermediate language. This intermediate language (IL) is interpreted bycompilers to produce output in a particular target language. As such,the IL is a superset of all possible output languages.

In some cases a particular target language may not support a concept orfunction represented by the IL in which case that compiler won't be ableto produce an output and will generate an error. The system willidentify filters that are not supported by the target indicated by theuser so the user knows in advance not to use these filters.

A large set of filters representing all of the common mathematicalfunctions and operations will be supplied with The system, although auser can extend this set of filters by implementing their own filters.The implementation of a shader filter requires that all input propertiesare identified and that the IL code fragment representing the operationof the filter is provided. The user will not have to provide code in theIL format but instead can provide code in some known existing formatsuch as HLSL or Cg. Internally The system will convert the code providedby the user into IL code to be stored with the filter. The filter willalso retain the original non-IL code the user provided so if the userlater wants to modify that code they can modify the original and Thesystem will reconvert it to IL.

Given a shader filter with its IL representation, it is possible thatthe same result can be expressed by a subgraph of filters with simplerIL representations. The system will be able to decompile IL into such asubgraph if one exists. This allows a user to provide a block of codefor a filter and have The system turn that code into a graph comprisedof primitive filters which the user (or another user) could modifygraphically by wiring to additional filters or variables.

The shader graph is a passive, representational wire graph that isinterpreted by a shader graph compiler to produce code in a particularoutput language (such as HSLS or Cg for example). Different compilersthat each produce output in a different format can interpret the sameshader graph. As new languages emerge, new compilers can be written toadd support for these new languages. Because a universal IL defines theshader filter's operation, the shader graph is not tied to anyparticular language.

The interpretation of a shader graph by a compiler can be broken outinto the following steps. A shader graph can define multiple methods forachieving a particular look called techniques. The client of the shadercan decide which technique to use based on different criteria. It iscommon to create different techniques that support different levels ofhardware compatibility. In this case one technique may refer to a morecomplex shader program that requires modern hardware to support it whileanother technique refers to a shader program with less complexity, butis supported on a wider range of hardware.

Within a particular technique, there may be multiple shader programsexecuted in individual passes. This allows a shader program thatutilizes all the resources of the hardware (for example the maximumnumber of supported simultaneous textures) to execute in one passfollowed by another shader program that also has all the resources ofthe hardware available to it in another pass.

As shown in the exemplary embodiment of FIG. 5, techniques and passesare represented in the shader graph by filters that combine inputs toproduce an output technique list. A pass filter combines the pixel andvertex shader trees to produce a pass structure. A technique filtercombines a collection of passes (any number of passes) to produce atechnique structure. Finally, a technique list filter combines acollection of techniques (any number of techniques) to produce atechnique list. The technique list, as indicated in FIG. 6, is the datathat is given to the compiler filter as input from which the compilerwill generate the output shader program code. Given the technique list,the first step of the compiler is to identify all the techniques andpasses and generate the appropriate declarations.

Uniform variables are global variables that can be referred to by anyfilter in any of the shader graphs. Their values will be set by thesoftware hosting the shader program and remain constant while the shaderprogram is executed. The compiler traverses all the shader trees in thegiven technique list and identifies these variables so that it can makethe appropriate declarations for them. It also stores descriptorsdescribing each uniform input and it provides these descriptors to thematerial filter. The material filter matches up the uniform variableswith the material's properties and uses the values of those propertiesto provide a value for the uniform variables. Some uniform variables arenot associated with a property on the material but instead derive theirvalue from a predefined source such as world transform, viewer position,etc. The user identifies uniform variables that are to be mapped to oneof these known quantities in advance by setting a tag called a semantic.Uniform variables for which the user has not associated a semantic areassumed to have matching material properties from which they can derivetheir values.

Each pixel or vertex shader tree referred to by the various passes isidentified and code in the target output language is generated. Thisinvolves identifying the varying inputs to the shader program andgenerating the appropriate function declaration. The shader outputfilter describes the data structure computed by the shader function(each input property of the output filter is a data member in thisstructure). This output structure is also declared at this stage.

The body of the code of the shader function is generated by recursivelytraversing the filter graph representing that shader. For each filter,the IL fragment associated with that filter is interpreted and convertedinto the target language. The IL fragment refers to the input propertiesof the filter and therefore code to compute those values must begenerated as well. To do this, a local variable is declared in theoutput code to hold the result and code to compute that result isgenerated. If the input property in question is wired to another filter,code is recursively generated that computes the result of that filter.If the input property is set to a constant value, then that value can beplaced in the output code where it is referenced.

As can be understood from the alternative illustrations of FIGS. 7 and8, the compiler can be optimized in a couple of ways. If multiple inputsto filters are wired to the same subgraph, code that produces the resultof that subgraph only needs to be generated and output in the finalprogram once. This avoids creating a program that generates the samevalue twice. Similarly, two individual subgraphs may be equivalent andreferred to by two separate inputs. In this case it would also beoptimal to only generate and output the code to produce the valuerepresented by the equivalent subgraphs once. The compiler in The systemwill detect these cases and generate optimized code. This leaves theuser free to construct the shader graph without concern for theseoperations.

The system application provides a workspace in which the user can createshaders. Within The system, the user can specify object, lights, andcameras to be used to render an image showing the shader in use in thecontext of a scene. While a user may ultimately export the output codeproduced by the shader compiler for use within another application orrendering environment, The system provides an environment to host theshader program while it is in development by the user. This allows theuser real-time interactive feedback as they make changes to the shaderprogram.

The material structure in The system previously described allows theuser to construct a scene wire graph that produces values used to setthe value of uniform input variables. A series of scene graph filterscould be wired together that form a compositing tree for a texture mapfor example, the result of which is used as a texture uniform variablein the shader program. The scene graph is part of the environmentprovided by The system to host the shader program, but is not convertedinto shader code as part of the output shader program. For example, theexemplary embodiment of FIG. 9 depicts a subgraph, that is part of thescene wire graph, that computes a texture to be used by the material.

The shader compiler generates code in a target language that can beexported for use in other systems, but in addition the shader compiler(encapsulated in the shader assembly) can by used inside the systemapplication. A renderable object filter, depicted in the exemplaryembodiment of FIG. 10, combines the elements needed to render a piece ofgeometry into a data structure given to the renderer. These elementsinclude geometry, a world space transformation matrix, and a structurerepresenting the shader program provided by the material filter calledan effect.

The effect structure encapsulates the code produced by the shader treecompiler along with a description of the uniform input variables forwhich values need to be provided before rendering with the effect. Thematerial filter also associates properties with some of these uniforminputs and stores this association with the effect structure.

The render is given a scene that contains a number of renderable objectfilters. The render evaluates each filter to get the geometry, worldspace transform, and effect structure. The effect structure contains thecode in a target language format generated by the shader tree compilerand the renderer gives this code to another third party compiler forthat target language (such as the Cg compiler or Microsoft's HLSLcompiler built into DirectX). The result is a shader program compiledfor the hardware on the user's machine that allows the shader to beviewed.

The renderer is also responsible for setting the values of the uniformvariables using APIs provided by the host rendering system (such asDirect X). The render retrieves the values for these variables fromproperties in the case where the material has associated properties withthe uniform inputs. In the case where a semantic was used to associate aknown parameter, the render provides the value for that parameter.

The result is that an image is rendered interactively that representsthe current state of the shader program as the user builds it. The useris able to preview the shader using any target language as long as ashader tree compiler and render API is provided that supports thatlanguage. The system will initially support compilers for HLSL and Cgand a renderer (Direct X and OpenGL) that supports shader programswritten in either of those languages.

The system is a software application that allows a user to construct agraphical representation of a shader program. The representationalshader graph is independent of any particular shader language howeverdifferent compilers that are part of The system can interpret that graphand produce a shader program in the format of a particular targetshading language. As new languages emerge, new compilers can be pluggedinto The system to support these new languages.

The shader graph is kept independent of any particular shader languagebecause the atomic units of functionality, the shader filters, representtheir corresponding operation in an intermediate language that is asuperset of all languages.

The system, in accord with the invention, defines an application whichprovides an interactive environment in which users can preview theirshaders in real-time to get immediate feedback as they construct them.The combination of the ease of use of the graphical representation, theability to collapse complexity into units of functionality (assemblies),and the ability to get real-time interactive feedback makes the creationof shaders accessible to those users who previously didn't have thetechnical ability to write shader code and provides an efficient andpowerful authoring environment to those with more technical ability.

1. A computer system for defining a representation of a shader ingraphical form, the system comprising: a first collection of graphicalobjects, each object having at least one input and at least one output,each of the first graphical objects defining a graphical function; asecond collection of graphical objects, each object having at least oneinput and at least one output, each of the second graphical objectsdefining a graphical parameter; a set of directed wires, a wire of theset coupling between at least one output of the second graphical objectset and at least one input of the first graphical object set; a codedefiner operative to express a wire coupling between a first and secondgraphical object, the code definer further operative to identify eachgraphical parameter and graphical function coupled by the wire; and acompiler, operative to translate and compile code defined by the definerinto a selected program language suitable for operating and controllinga hardware shader.
 2. In a computer system, a method for defining arepresentation of a shader in graphical form, the method comprising:defining first collection of graphical objects, each object having atleast one input and at least one output, each of the first graphicalobjects defining a graphical function; defining second collection ofgraphical objects, each object having at least one input and at leastone output, each of the second graphical objects defining a graphicalparameter; coupling a directed wire between at least one output of thesecond graphical object set and at least one input of the firstgraphical object set; expressing a wire coupling between a first andsecond graphical object in a representational software language;expressing each graphical parameter and graphical function coupled bythe wire in said representational language; and translating andcompiling the representational software language into a selected programlanguage suitable for operating and controlling a hardware shader.